Dependency Analysis of Scrambled References for Better Evaluation of Japanese Translation

In English-to-Japanese translation, BLEU (Papineni et al., 2002), the de facto standard evaluation metric for machine translation (MT), has very weak correlation with human judgments (Goto et al., 2011; Goto et al., 2013). Therefore, RIBES (Isozaki et al., 2010; Hirao et al., 2014) was proposed. RIBES measures similarity of the word order of a machine-translated sentence and that of a corresponding human-translated reference sentence. RIBES has much stronger correlation than BLEU but most Japanese sentences have alternative word orders (scrambling), and one reference sentence is not sufficient for fair evaluation. Isozaki et al. (2014) proposed a solution to this problem. This solution generates semantically equivalent word orders of reference sentences. Automatically generated word orders are sometimes incomprehensible or misleading, and they introduced a heuristic rule that filters out such bad sentences. However, their rule is too conservative and generated alternative word orders for only 30% of reference sentences. In this paper, we present a rule-free method that uses a dependency parser to check scrambled sentences and generated alternatives for 80% of sentences. The experimental results show that our method improves sentence-level correlation with human judgments. In addition, strong system-level correlation of single reference RIBES is not damaged very much. We expect this method can be applied to other languages such as German, Korean, ∗This work was done while the second author was a graduate student of Okayama Prefectural University. Spearman’s ρ with adequacy NTCIR-7 JE RIBES JE BLEU NTCIR-9 JE RIBES JE BLEU EJ RIBES EJ BLEU NTCIR-10 JE RIBES JE BLEU EJ RIBES EJ BLEU 0.0 0.2 0.4 0.6 0.8 1.0 Figure 1: RIBES has better correlation with adequacy than BLEU (system-level correlation) Turkish, Hindi, etc.